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LONG-TERM  GOALS 

We  seek  to  develop  radiation  parameterizations  for  Navy  models  that  are  computationally  efficient  and 
work  seamlessly  across  models  at  all  time  and  space  scales,  especially  from  regional  models  to  global 
models. 

OBJECTIVES 

We  are  adapting  radiation  codes  developed  for  climate  models  for  use  in  the  Navy’s  global  weather 
forecast  model  (NOGAPS/NAVGEM)  and  limited  area  model  (COAMPS).  Our  long-temi  goal  is  to 
develop  codes  that  are  scale-aware,  computationally  efficient  across  a  range  of  computer  architectures, 
and  operate  continuously  rather  than  at  infrequent  “radiation  time  steps”. 

APPROACH 

We  have  developed  radiation  codes  known  as  “PSrad”  which  are  modeled  on  the  RRTMG 
parametization  (Mlawer  et  al.  1997;  Iacono  et  al.  2008).  We  make  use  of  the  RRTMG  description  of 
gas  optics,  which  is  among  the  most  accurate  parameterizations  available  (Oreopoulos  et  al.  2012)  and 
initially  make  many  of  the  same  algorithmic  decisions,  including  the  choice  to  neglect  longwave 
scattering.  Our  codes  are  intended  as  a  drop-in  replacement  for  RRTMG  (which  has  already  been 
implemented  in  Navy  forecast  models  by  Ming  Liu)  but  we  have  implemented  it  almost  entirely  from 
scratch.  The  most  important  technical  difference  lies  in  the  organization:  we  have  made  the  code 
substantially  more  modular,  and  each  of  our  subroutines  is  designed  to  operate  on  many  columns  at  a 
time,  a  choice  that  increases  computational  efficiency  on  a  wide  range  of  platfonns.  Operational 
centers  such  as  the  European  Centre  for  Medium-Range  Weather  Forecasts  have  often  modified 
RRTMG  in  this  way  (Morcrette  et  al.  2008). 

Sub-grid  scale  variability  is  treated  using  “sub-columns”  (Raisanen  et  al.  2004;  Pincus  et  al.  2006): 
discrete  random  samples,  each  treated  as  internally  homogeneous,  that  are  consistent  with  the 
distributions  of  possible  cloud  states  within  each  column,  including  fractional  cloudiness  in  each  layer 
and  assumptions  about  the  vertical  correlations  between  layers  (so-called  “cloud  overlap”).  This 
treatment  is  a  generalization  of  the  Monte  Carlo  Independent  Pixel  Approximation  (Pincus  et  al.  2003). 
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The  main  functional  innovation  in  PSrad  is  support  for  a  range  of  choices  for  spectral  sampling, 
including  broadband  integration  (most  likely  applied  at  relatively  infrequent  “radiation  time  steps”) 
amd  a  finite  number  of  pre-determined  “leagues”  of  g-point  teams  constructed  to  limit  the  error  in 
surface  fluxes,  as  described  in  Pincus  and  Stevens  (2013). 

WORK  COMPLETED 

The  initial  version  of  PSrad  is  now  complete,  thorougly  tested  and  debugged,  is  functioning  as  the 
radiation  scheme  in  the  climate  model  ECHAM  6.2  developed  at  the  Max  Planck  Institute  for 
Meteorology.  Implementation  in  ECHAM  exposed  the  code  to  a  wider  range  of  critical  eyes  and 
identified  some  bugs  and  other  inconsistentices.  The  code  has  been  somewhat  refined  for 
computational  perfonnance  and  portability.  We  built  a  “fast  math  library”  module,  for  example,  that 
encapsulates  some  of  the  frequently-used  calls  to  functions  like  the  exponential;  compiler  directives 
allow  for  highly-optimized  vendor-supplied  vector  versions  to  be  used  where  available.  We  continue  to 
optimize  some  of  our  numerics,  especially  those  related  to  the  two-stream  calculations  required  for 
shortwave  fluxes. 

Despite  these  optimiziations,  PSrad  is  2-3  times  slower  (depending  on  the  platform)  than  the  RRTMG 
code  from  which  it  descends  when  doing  broadband  integration  (i.e.  all  spectral  intervals).  The 
majority  of  this  slowdown  is  due  to  the  calculation  of  gas  absorption  coefficients  from  temperatures, 
pressures,  and  gas  concentrations.  RRTMG’ s  calculations  are  efficient  because  the  same  spectral 
quadrature  point  is  used  by  all  columns  and  each  point  is  done  in  order,  where  PSrad  makes  neither 
assumption.  We  continue  to  explore  ways  to  reduce  this  disparity  in  computational  cost. 

About  eighteen  months  ago  ECMWF  approached  us  about  a  collaboration  to  assess  the  utility  of  the 
spectral  sampling  approach  in  reducing  certain  temperature  errors  due  to  complicated  terrain.  This 
collaboration  is  slowly  ongoing. 

RESULTS 

The  goal  of  PSrad  was  not  to  reproduce  the  results  of  RRTMG  but  rather  to  explore  the  impact  of  more 
frequent  but  noisy  estimates  of  radiation  on  the  accuracy  of  model  forecasts.  We  took  two  approaches. 
The  first  treats  the  parameterization  problem  abstractly  and  assess  the  degree  to  which  various 
radiative  transfer  approximations,  including  different  temporal  frequencies  of  broadband  calculations 
and  various  sets  of  g-point  teams,  affect  the  evolution  of  the  model  as  compared  to  a  reference 
forecasts  that  computes  the  complete  radiation  field  at  every  time  step.  We  did  this  using  a  29-member 
ensemble  of  “perfect-model”  forecasts  with  ECHAM.  Figure  1  (from  Pincus  and  Stevens,  2013)  shows 
results  with  respect  to  the  global  distribution  of  2  m  air  temperature  at  T63  resolution,  averaged  over 
the  first  ten  days.  Root-mean-square  differences  between  any  set  of  forecasts  are  finite  because  all 
approximations  use  discrete  samples  to  represent  cloudiness  and  so  diverge  over  time;  this  limit  is 
shown  in  the  pink  line.  RMS  errors  for  the  various  approximations  are  plotted  as  a  rough  function  of 
computation  time.  Our  interpretation  of  this  figure  is  that  the  divergence  of  an  ensemble  from  the 
reference  ensemble  is  primarily  controlled  by  the  amount  of  computational  effort  spent,  regardless  as 
to  whether  this  is  spent  on  broadband  calculations  of  variable  frequency  (purple  dots)  or  very  frequent 
calculations  with  g-point  teams  of  various  sizes  (green  dots). 

We  are  also  assessing  the  impact  of  the  sampling  approximations  on  temperature  errors  in  real 
forecasts  with  ECMWF.  Here  error  is  measured  with  respect  to  point  SYNOP  measurements  as 
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opposed  to  gridded  analyses.  ECMWF  has  explored  single  summer  and  winter  time  months  and  find 
smaller  errors  at  some  stations,  worse  errors  at  others,  and  no  statiically  significant  change  at  most 
stations,  indicating  that  errors  in  most  places  are  not  primarily  driven  by  radiation  errors.  We  are 
working  with  the  Centre  to  design  experiments  to  isolate  the  impact  of  the  radiation  approximation  in  a 
statisically  robust  way.  We  are  particularly  interested  in  the  behavior  of  the  error  across  spatial  scales, 
since  most  centers  including  FNOC  can  not  afford  the  high  spatial  resolution  (T1279  -  T1999)  used  at 


ECMWF. 
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Figure  1:  Time-mean  error  versus  a  rough  measure  of  computational  cost  for  two  methods  of 
coupling  radiation  and  dynamics  in  a  global  model.  Broadband  computations  applied  sparsely  in 
time  are  shown  in  purple  and  the  “teams”  of  spectral  intervals  are  in  green.  Error  is  measured  as 
the  mean  over  10  days  of  the  global  RMS  difference  in  2  m  air  temperature  relative  to  a  reference 
forecast,  and  computational  cost  as  the  daily  number  of  calls  to  the  shortwave  solver  (which 
dominates  the  overall  cost).  The  pink  line  shows  the  minimum  achievable  error  (i.e.,  the  error 
introduced  by  another  realization  of  cloud  states  sampled  in  the  Monte  Carlo  independent  pixel 
approximation).  Approximation  errors  for  the  two  sampling  strategies  are  commensurate 

for  a  given  computational  cost. 


IMPACT/APPLICATIONS 

As  mentioned  above,  the  very  general  approach  to  spectral  integration  taken  by  PSrad  means  that  the 
code  is  about  twice  as  slow  on  some  architectures  as  the  RRTMG  code  from  which  it  is  dervied  when 
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computing  broadband  fluxes.  Because  RRTMG  is  already  implemented  in  NAVGEM,  the  desirability 
of  implementing  PSrad  depends  in  part  on  how  important  it  seems  to  use  the  capability  to  replace 
infrequent  broadband  calculations  with  more  frequent  spectral  samples.  Initial  perfect  model  results 
with  ECHAM  do  not  suggest  a  compelling  practical  reason.  This  may  change  when  model  error  is 
accounted  for  (i.e.  when  making  real  forecasts)  and  we  are  exploring  this  possibilty  though  our 
collaboration  with  ECMWF.  But  in  the  short  term  there  seems  to  be  no  compelling  reason  to  expend 
the  effort  to  replace  RRTMG  with  PSrad  in  NAVGEM.  Instead  we  propose  to  focus  efforts  on  the 
regional  model  COAMPS,  where  smaller  grid  sizes  and  shorter  time  steps  will  allow  for  greater 
flexibity  with  respect  to  radiation  choices.  (Access  to  COAMPS  code  is  also  more  open  than  has 
historically  been  the  case  with  NAVGEM,  and  this  will  also  facilitate  collaboration.) 

RELATED  PROJECTS 

The  lessons  learned  during  the  development  of  PSrad  were  followed  closely  by  other  developers 
including  those  responsible  for  RRTMG.  To  implement  bring  these  ideas  back  to  RRTMG  AER, 
University  of  Colorado,  and  computational  scientists  from  NCAR  developed  a  proposal  to  National 
Oceanographic  Parternship  Program  (NOPP)  to  develop  high-performance  versions  of  RRTMG  that 
incorporate  the  functionality  of  PSrad.  These  codes  will  be  structured  to  work  efficiently  on  all 
platforms  including  emerging  heterogenous  architectures  such  as  Graphic  Processing  Units  and 
Multiple  Independent  Cores.  We  have  been  told  that  this  propsal  will  be  funded  but  have  not  received 
the  award  yet.  We  expect  the  next  revision  of  RRTMG  to  replace  PSrad. 
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